In this project, you’re going to take a peek into the realm of neural network machine translation. You’ll be training a sequence to sequence model on a dataset of English and French sentences that can translate new sentences from English to French.
Since translating the whole language of English to French will take lots of time to train, we have provided you with a small portion of the English corpus.
在这个项目中，您将要窥探神经网络机器翻译的领域。 您将在英文和法文句子的数据集上训练一个序列来对序列模型进行排序，这可以将新句子从英语翻译成法语。
因为将英语的整个语言翻译成法语将需要很多时间来训练，我们为您提供了英文语料库的一小部分。
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import helper
import problem_unittests as tests
source_path = 'data/small_vocab_en'
target_path = 'data/small_vocab_fr'
source_text = helper.load_data(source_path)
target_text = helper.load_data(target_path)
Play around with view_sentence_range to view different parts of the data.
view_sentence_range = (0, 10)
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import numpy as np
print('Dataset Stats')
print('Roughly the number of unique words: {}'.format(len({word: None for word in source_text.split()})))
sentences = source_text.split('\n')
word_counts = [len(sentence.split()) for sentence in sentences]
print('Number of sentences: {}'.format(len(sentences)))
print('Average number of words in a sentence: {}'.format(np.average(word_counts)))
print()
print('English sentences {} to {}:'.format(*view_sentence_range))
print('\n'.join(source_text.split('\n')[view_sentence_range[0]:view_sentence_range[1]]))
print()
print('French sentences {} to {}:'.format(*view_sentence_range))
print('\n'.join(target_text.split('\n')[view_sentence_range[0]:view_sentence_range[1]]))
As you did with other RNNs, you must turn the text into a number so the computer can understand it. In the function text_to_ids(), you'll turn source_text and target_text from words to ids. However, you need to add the <EOS> word id at the end of target_text. This will help the neural network predict when the sentence should end.
You can get the <EOS> word id by doing:
target_vocab_to_int['<EOS>']
You can get other word ids using source_vocab_to_int and target_vocab_to_int.
与其他RNN一样，您必须将文本转换成数字，以便计算机能够理解。 在函数text_to_ids（）中，你可以将source_text和target_text从单词转成ids。 但是，您需要在target_text'的末尾添加
您可以通过以下方式获取“
蟒蛇
target_vocab_to_int[ '<EOS>']您可以使用“source_vocab_to_int”和“target_vocab_to_int”获取其他单词id。
def text_to_ids(source_text, target_text, source_vocab_to_int, target_vocab_to_int):
"""
将源文本和目标文本转换为正确的字ids
：param source_text：包含所有源文本的字符串。
：param target_text：包含所有目标文本的字符串。
：param source_vocab_to_int：从源字词到id的字典
：param target_vocab_to_int：从目标词到id的字典
：return：列表元组（source_id_text，target_id_text）
"""
#定义源文件ID
source_text_id = []
#使用换行分割文本
for sentence in source_text.split("\n"):
#使用空格分割句子，转换成词典中某个单词对应的数字
sentence_id =[source_vocab_to_int[word] for word in sentence.split()]
source_text_id.append(sentence_id)
#和上面的实现一样
target_text_id=[]
for sentence in target_text.split("\n"):
sentence_id =[target_vocab_to_int[word] for word in sentence.split()]+ [target_vocab_to_int['<EOS>']]
target_text_id.append(sentence_id)
print (source_text_id)
print (target_text_id)
return (source_text_id, target_text_id)
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_text_to_ids(text_to_ids)
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
helper.preprocess_and_save_data(source_path, target_path, text_to_ids)